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SPECIFICATION 
TITLE 

0001 METHOD AND SYSTEM FOR INTERACTIVE REGION SEGMENTATION 

CROSS-REFERENCE TO RELATED APPUCATIONS 

0002 The present invention shares the same inventors and assignee as, and is related to, 
the following ^plications: 

0003 Docket number pp-02-0 1 -2003, entitled "METHOD AND SYSTEM FOR 
INTERACTIVE CROPPING OF A GRAPHICAL OBJECT WITHIN A 
CONTAINING REGION", as well as Docket number pp-01-01-2003, entitled 
"METHOD AND SYSTEM FOR DISTRIBUTING MULTIPLE DRAGGED 
OBJECTS", in that the methods of object cropping and object distribution described 
therein may optionally be combined with the user interface elements described herein. 

0004 The present application formalizes the provisional utility patent application with 
application number 60-446,751 and confirmation number 4660, filed February 3 2003. 

FIELD OF THE INVENTION 

0005 The present invention relates particularly to a graphics imaging system for 
producing visual designs and more particularly to a graphics imaging system that 
presents a display of a plurality of regions arranged into a regular structure, for 
example to form a collage of bitmapped images. 

BACKGROUND OF THE INVENTION 

0006 Computer systems are sometimes used to process digital imagery, such as that 
captured by digital cameras and scanners, into new forms. The processing may include 
assembling several images into new images, such as mosaics or collages. The present 
invention includes a method for streamlining the segmentation of a region or volume 
into smaller regions or volumes. While the examples provided refer to the assembly of 
collages from multiple source images, the technique is applicable to any task involving 
the arrangement of multiple two- or three-dimensional regions. 
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SUMMARY OF THE INVENTION 

0007 The present invention provides for an interaction technique, implementable on a computer 

readable medium, for interactively segmenting a region into an array of smaller regions. We use 
the term *region* to indicate a two- or three-dimensional bounded area on a computer display 
which can be used, without lack of generality, to contain a bitmapped raster image. 



0008 In one aspect of the invention, the user enters a segmenting mode, interactively segments a 
particular region into a plurality of smaller regions which in aggregate retain the overall 
dimensions of the initial region, and then leaves the segmenting mode. 

0009 In another aspect of the invention, the segmentation is performed by direct manipulation 
of a control drawn on the border of said initial region with a user inter&ce device such as a 
mouse or keyboard. 

00 1 0 In another aspect of the invention, the segmentation is computed using the updated 
position of a pointing device such as a mouse relative to the position at the start of interaction. 
A keyboard may also be used. The segmentation forms a combination of rows and/or columns 
of evenly-spaced regions which fiU the initial region when the user exits the segmentation 
mode. 

001 1 In another aspect of the invention, the user receives visual feedback during segmentation 
which indicates the number and position of subregions which will be created when the 
segmentation mode is exited. This feedback may take the form of lines on the original region at 
the actual location where new regions will be created, or it may take the form of an overlaid 
grid of cells indicating the number of new regions that will be formed. 

0012 In another aspect of the invention, the original region may either be replaced by the 
collection of new regions at the end of interaction, or it may continue to exist as a container of 
the new regions in a parent-child relationship. 

BRIEF DESCRIPTION OF THE DRAWINGS 

0013 A specific embodiment of the invention will now be described, by way of example, with 
reference to the accompanying drawings in which: 

• FIG. 1 depicts a computer system suitable for implementing a system for segmenting 
regions, according to the present invention. 

• FIG. 2 depicts an abstract collage, demonstrating various operations that can be 
£^piied to the regions it contains. 

• FIG. 3 depicts a simple, grid-based tiled collage. 

• FIG. 4 depicts a non-grid collage. 

• FIG. 5 depicts the relationship between regions and bitmapped source images they 
may contain 
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• FIG. 6 depicts an example set of manipulators that decorate the frame of a region in 
the preferred embodiment of the invention. 

• FIG. 7 depicts region rotation using a manipulator handle. 

• FIG. 8 depicts region resizing using a manipulator handle. 

• FIG. 9 depicts image panning using a manipulator handle. 

• FIG. 10 d^icts image cropping using a manipulator handle. 

• FIG. 1 1 depicts region deletion using a manipulator handle. 

• FIG. 12 depicts a simple one-step region segmentation. 

• FIG. 13 depicts the creation of a complex collage creation using a sequence of region 
segmentations. 

• FIG. 14 depicts the steps of region segmentation in the preferred mouse-based 
implementation of the invention. 

• FIG. IS draws attention to the visual references drawn over the region during mouse- 
based segmentation. 

• FIG. 16 depicts alternatives to said visual references drawn on overlapped regions. 

• FIG. 1 7 depicts an alternative embodiment of the visual references drawn during 
mouse-based segmentation. 

DETAILED DESCRIPTION 

0014 In the following discussion, the present invention is described for illustrative purposes 

with reference to the editing of raster image information. However, one of ordinary skill in the 
art will recognize that the invention, in its broadest aspect, is applicable to processes other than 
image manipulation, and it is not intended that the scope of the invention be so limited. For 
example, the present invention is also applicable to the editing of video data, and to media data 
in general. 

IMAGING SYSTEM 

A computer graphics imaging system 1 is schematically depicted in FIG. 1. The graphics 
imaging system 1 includes a computer 2 that has a central processing unit (CPU) 3 which may 
include local memory 3a, static memory 4 such as Read-only memory (ROM), main memory S 
such as Random Access memory (RAM), mass memory 6 such as a computer disk drive, a 
system bus 7, adaptor(s) for external input devices 8, and a display adapter 9 which may include 
local memory 9a. The computer 2 communicates with an alphanumeric inpvX device 10 such as 
a keyboard, a pointer device 1 1 such as a mouse for manipulating a cursor and making 
selections of data via said input adapter 8. The computer 2 conmiunicates with a video display 
12 such as a computer monitor via said display adapter 9. 



COMPUTER 

0015 
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0016 The computer 2 executes imaging software described below to allow the system 1 to 
render high quality graphics images on the monitor 12. The CPU 3 comprises a suitable 
processing device such as a microprocessor, for example, and may comprise a plurality of 
suitable processing devices. The gnq)hics adaptor 9 may also be cq>able of executing 
instructions. Instructions are stored in the CPU local memory 3a, static memory 4, display 
adapter local memory 9a, main memory 5, and/or mass memory 6 and are executed by the CPU 
3 or the display adapter 9. 

00 1 7 The static memory 4 may comprise read only memory (ROM) or any other suitable 
memory device. The local memory may store, for example, a boot program for execution by 
CPU 3 to initialize the graphics imaging system 1 . The main memory 5 may comprise random 
access memory (RAM) or any other suitable memory device. The mass memory 6 may include 
a hard disk device, a floppy disk, an optical disk, a flash memory device, a CDROM, a file 
server device or any otfier suitable memory device. For this detailed description, the term 
memory comprises a single memory device and any combination of suitable devices for the 
storage of data and instructions. 

0018 The system bus 7 provides for the transfer of digital information between the hardware 
devices of the graphics imaging system 1 . The CPU 3 also receives data over the system bus 7 
that is input by a user through alphanumeric input device 10 and/or the pointer device 1 1 via an 
input adaptor 8. The alphanumeric input device 10 may comprise a keyboard, for example, that 
comprises alphanumeric keys. The alphanumeric input device 10 may comprise other suitable 
keys such as fimction keys for example. The pomter device 1 1 may comprise a mouse, track- 
ball, tablet and/or joystick, for example, for controlling the movement of a cursor displayed on 
the computer display 12. 

0019 The graphics imaging system 1 of FIG. 1 also includes display adapter hardware 9 that 
may be implemented as a circuit that interfaces with system bus 7 for facilitating rendering of 
images on the computer display 12. The display adapter hardware 9 may, for example, be 
implemented with a special graphics processor printed circuit board including dedicated 
random access memory 9a that helps speed the rendering of high resolution, color unages on a 
viewing screen of the display 12. 

0020 The display 12 may comprise a cathode ray tube (CRT) or a liquid crystal display 
particularly suited for displaying graphics on its viewing screen. The invention can be 
implemented using high-speed graphics workstations as well as personal computers having one 
or more high-speed processors. 



Specification - Page S of 16 



0021 The graphics imaging system 1 utilizes specialized graphics software particularly suited to 

take advantage of the imaging hardware included in the display system 1 depicted in FIG. 1. 
The software implements a user inter&ce and related processing algorithms as described in 
subsequent sections to enable the user to produce graphical works viewed on the display 12 and 
which may be stored in mass memory 6. The graphical works may involve the assembly of 
graphical objects such as bitmaps. Source material for use with such a system can include 
previously digitized materials stored on a computer memory 6 such as images from digital 
cameras, scanning devices, or the internet, stored on a large capacity hard or fixed disk storage 
device. 



COLLAGES 

0022 



For the purposes of this invention a Collage is defined as an arrangement of two- 
dimensional graphical objects, such as digital images, into a larger graphical object. As shown 
in FIG. 2, the regions 21a-g within the collage 20 may tile (21a, 21b, 21c), overlap (2 Id, 21e), 
be rotated to any angle (2 1 f), skewed (2 1 g), or otherwise affected by the user. 



0023 Collages which fit neatly together, with even spaces between adjacent images, are visually 
pleasing. As shown in FIG. 3, regular grids, such as 2x2 or 3x4 arrangements of four or twelve 
images respectively, are often used to assemble a composite image from several source images. 

0024 Users who wish collages of more visual complexity may choose to arrange the tiled 
images into non-grid patterns. As shown in FIG. 4, these collages may form a Tjrick wall' 
interleaving or other more sophisticated arrangements. 

0025 The current invention provides the user with tools for quickly constructing both regular 
grid and non-grid collages. 



REGIONS 

0026 



The collage is composed of a set of user-manipulable frames that define regions within the 
collage. As shown in FIG. 5, each region 21 frames and contains exactly one graphical object 
31, such as a bitmapped image, which can be panned and zoomed within the frame. The 
graphical object is a reference to a source object 32, such as a bitmap. 



0027 As shown in FIG. 6, the user can apply several types of manipulation to a frame by 

manipulating controls (33-38) which in the preferred embodiment of the invention are built into 
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the frame of each region. These controls can be driven via a user interface device such as a 
mouse, or using a keyboard. The operations include: 

• Rotating the region 2 1 (see FIG. 7) 

• Resizing the region 2 1 (see FIG. 8) 

• Panning the unage 3 1 contained within the region 21 (see FIG, 9) 

• Cropping the image 3 1 contained within the region 2 1 (see FIG. 1 0) 

• Deleting the region 2 1 (see FIG. 1 1 ) 

• Segmenting the region 2 1 (described below). 



REGION SEGMENTATION 

0028 The present invention specifically relates to the segmentation, or splitting, of regions: the 

production of a set of smaller regions which in aggregate retain the overall dimensions of the 
initial region. 



0029 Segmentation is a useful tool for building collages: simple grid-based collage layouts can 
be constructed in a single operation (FIG. 12) while complex, irregular collages can be 
constructed by applying sequential segmentation operations (FIG. 13) 

0030 Referring again to FIG. 6, there is shown, in graphical form, a region 2 1 . In one 
embodiment of this invention, said region is decorated with interactive manipulator handles 33- 
38 enabling the user to apply various operations on the region and the image it contains via a 
pointing user interfoce device. In particular there exists a handle 33 v^ch enables segmentation 
of said region 2 1 . Said handle may appear anywhere in or about said region 2 1 . 

SEGMENTING REGIONS USING A MOUSE INPUT DEVICE 

003 1 In one embodiment of this invention, the user segments the region by manipulating an 
input device such as a mouse as illustrated in FIG. 14: 



1 . Moving the mouse such that the pointer 40 is positioned over the split handle 33 

• The handle 33 may assume a new visual state, such as a brightened or highlighted 
representation, to indicate that it will become active if the user presses the mouse 
button. 

2. Pressing the mouse button 

• The handle 33 and other decoration on the frame may disappear to allow the user 
to concentrate on the image itself 

3 . Dragging the mouse down and to the right 
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• As the user drags, the display may update to reflect the segmentation the user is 
currently specifying. For example, it may siq>erimpose lines 41 on the region 
indicating the number of splits that will occur 

• rte Algorithm for calculating the number of splits and alternative visual 
representations are discussed below, 

4. Releasing the mouse button 

• The region will be segmented if it has not already done so 

• The handle 33 and all other decoration on the frame may reappear if they were 
hidden in step 2. 

• The handles may be drawn smaller if necessary to fit within the new regions 

0032 In step 3 above, the number of rows and colunms that the region(s) will be segmented into 

is calculated by determining the distance from the current pointer position to the position at the 
time of the button press in step 2. This distance is composed of an X and a Y coordinate: the 
segmentation is computed as follows: 

void CPiclnfoManager::splitFrameCCPicInfo* in.ppi, LONG in^xcells, LONG in_ycens) 
{ 

ASSERT(in_ppi); 

// Set the rotated box based on the rotation of the frame 
RectF rcBoxCin«ppi->GetFrameBox()) ; 
REAL f Angle = in_ppi->GetRotationO ; 
Matrix m; 

m.Trans1ate(rcBox.X, rcBox.Y); 

m.RotateAtCf Angle, Pol ntF(CrcBox. width / 2), (rcBox. Height / 2))); 

// Determine the final cell size 

rcBox. width /= in^xcells; 

rcBox. Height /= in_ycens; 

CPiclnfo* ppi = in_ppi; 

for (LONG y = 0; y < in.yCells; y++) 

{ 

for (LONG x = 0; X < in^xcells; x++) 
{ 

// If we're the upper left cell, then we're ourself 

if (X II y) 

{ 

ppi = CreateFrame (RectF (0, 0, 0, 0)); 

} 

// Resize and rotate the frame 

ppi ->set Framesi ze (rcBox . wi dth , rcBox . Hei ght) ; 

ppi ->SetRotati on (f Angle) ; 

// Rotate it within the final box 

Point F ptBox((x * rcBox. width) + (rcBox. width / 2), 

(y * rcBox. Height) + (rcBox. Height / 2)); 
m.TransformPoints(&ptBox) ; 
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// Set it's new rotated position and show the item 
ppi->SetFramePosCptBox.x-(rcBox, width / 2), pt Box. Y-CrcBox. Height / 

2)); 

ppi->ShowCtrue); 

} 

} 

} 

REGION SEGMENTATION MODE 

0033 Once the mouse button was pressed while the cmor 40 was over the split handle 33 in 

step 2 of the previous section, and until the mouse button was released in the subsequent step 4, 
the graphics system was in "Region Segmentation Mode". 



0034 In another embodiment of this invention, the user has other means for entering and leaving 

this mode for a given selected region(sX such as via the use of a keyboard: 

1 . Pressing and releasing a key (such as 's') to enter Region Segmentation Mode 

• (applying the segmentation using any technique described here) 

2. Subsequently pressing a key (such as 'enter*, or V again) to exit the mode 

or 

1 . Pressing and holding a key (such as V) to enter Region Segmentation Mode 

• (applying the segmentation using any technique described here) 

2. Subsequently releasing said key to exit the mode 



SEGMENTING REGIONS USING A KEYBOARD INPUT DEVICE 

003S In another embodiment of this invention, the user segments a selected region using a 

keyboard, as follows: 

1 . Entering the Region Segmentation Mode 

• (For example, by pressing and releasing the 's ' key) 

and then... 

2a). Pressing and releasing a key which increases horizontal segmentation (such as 
the right arrow key) 

• A column will be added to the previewed horizontal segmentation of the region 
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or 



2b). Pressing and releasing a key which decreases horizontal segmentation (such as 
the left arrow key) 

• A column will be removed from the previewed horizontal segmentation of the 
region 



or 



2c). Pressing and releasing a key which increases vertical segmentation (such as the 
down arrow key) 

• A row will be added to the previewed vertical segmentation of the region 



or 



2d), Pressing and releasing a key which decreases vertical segmentation (such as the 
up arrow key) 

• A row will be removed from the previewed vertical segmentation of the region 

ABORTING SEGMENTATION 

0036 In another embodiment of this invention, the user is able to abort the segmentation 
operation while in Region Segmentation Mode. This is accomplished by pressing a key, such as 
the 'esc' key. This removes any previewed segmentation, refreshes the display to include the 
handles decorating the regions again if necessary, and exits Region Segmentation Mode. 

VISUAL FEEDBACK DURING SEGMENTATION 

0037 In another embodiment of this invention, the user is provided with visual feedback as the 
segmentation operation progresses under his control. 

GRID LINES 

0038 In the preferred embodiment of the invention, and as illustrated in FIG. 14, feedback 
regarding the progress of the segmentation is provided to the user while in Region Subdivision 
Mode by means of a set of lines 41 drawn over the region(s) undergoing segmentation. The 
position of the lines indicates the actual location of row and/or column divisions that will result 
when Region Segmentation Mode is exited. 
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0039 As the user increases or decreases the amount of segmentation, these lines are updated to 

reflect the new position of the splits (FIG. 14). 



0040 These lines are drawn on top of the region(s) being subdivided; without loss of generality 
they may be drawn above or below other regions which happen to overlap said region(s) (FIG. 
15). 

0041 When Region Segmentation Mode is exited, the lines are removed. The new arrangement 
of regions resulting from the segmentation will exactly align with the final position of said 
lines. 



CELLS 

0042 In another embodiment of the invention as illustrated in FIG. 1 7, feedback regarding the 

progress of the segmentation is provided to the user while in Region Subdivision Mode by 
means of a grid of cells 41 drawn over the display, positioned at the location where interaction 
began. 



0043 As the user increases or decreases the amount of segmentation, the cells in the grid are 

highlighted 42 to reflect the number of rows and columns that wUl result froni the 
segmentation. 



0044 The grid may have a minimum dimension (say, 3 by 3 cells). In the event that the user 

selects a segmentation of greater than this minimum dimension, the grid will be extended in the 
appropriate direction to completely contain the number of cells required to continue to reflect 
the desired segmentation. 



0045 This grid of cells 4 1 is drawn on top of all regions on the display. 

0046 When Region Segmentation Mode is exited, the grid is removed. The new arrangement of 
regions resulting from the segmentation will reflect the number of cells there were highlighted 
during the region segmentation operation. 

COMPLETING SEGMENTATION 

0047 When the user exits Region Segmentation Mode without aborting, a set of new regions are 
added to the display such that their aggregate dimensions match the dhnensions of the original 
region. 
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0048 



In the preferred embodiment of the invention, the original region is replaced by the new 
regions which are independent and can be further manipulated or split by the user without 
affecting other regions. 



0049 In another embodiment of the invention, the original region is retained and the new 
regions are considered as subregions of the original region in a parent-child relationship. 

0050 In either case it is possible to retain the contents of the original frame, if any, within one of 
the newly-created regions. 

MODIFICATIONS AND ALTERNATE EMBODIMENTS 

0051 Having described the invention, it should be apparent to those of ordinary skill in the art 
that the foregoing is illustrative and not limiting. Numerous modifications, variations and 
alterations may be made to the described embodiments without departing from the scope of the 
invention by one of ordinary skill in the art and are contemplated as falling within the scope of 
the invention as defined by the appended claims. 
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